Messaging system interacting with dynamic extension app

ABSTRACT

A messaging system includes a first messaging app on a first device and a second messaging app, on a second device, that can exchange messages (e.g. text messages, etc.), and the second messaging on can transmit content from a second extension app that operates with the second messaging app to the first messaging app that operates with a first extension app. The content can be opaque to the messaging app, but can be processed by the extension apps to display live, dynamic content in message bubbles in a message transcript of a conversation.

This application is a continuation of U.S. application Ser. No.17/087,540, entitled “Messaging System Interacting With DynamicExtension App”, filed Nov. 2, 2020, which is a continuation of U.S.patent application Ser. No. 15/902,507, entitled “Messaging SystemInteracting with Dynamic Extension App”, filed on Feb. 22, 2018, whichclaims the benefit of U.S. Provisional Patent Application No.62/514,742, entitled “Messaging System Interacting with DynamicExtension App”, filed on Jun. 2, 2017, the disclosure of each of whichis hereby incorporated herein in its entirety.

BACKGROUND

The embodiments described in this disclosure relate to messaging systemssuch as text messaging systems on cellular telephones or othercommunication devices or data processing systems.

The use of text messaging systems began many years ago. For example,wireless cellular telephone carriers, such as Verizon or AT&T, allowedtext messages through the Short Message Service (SMS) for cell phones inthe 1990s before smartphones were available. Typically, the amount ofdata transmitted has been limited by rules established by the carriers.Recently, as the use of smartphones (e.g. iPhones) and tablet computers(e.g. iPad) has increased, the text messaging systems have developed theability to send images, such as photos or emojis. In addition, messagingsystems such as iMessage from Apple Inc. of Cupertino, Calif. haveallowed users to also send and receive text and images through “public”networks which include “public” WiFi access points and the Internet (inaddition to using the wireless carrier's private cellular telephonenetworks), and messaging systems such as iMessage can seamlesslytransition between the use of public and private networks depending onthe availability of, for example, WiFi access points or thecompatibility of the other user's device (which may not be compatiblewith iMessage).

SUMMARY OF THE DESCRIPTION

The embodiments described herein operate in a messaging system thatincludes a first messaging application or app on a first device and asecond messaging application, on a second device, that can exchangemessages such as text messages, etc. The first messaging app and thesecond messaging app can each be configured to operate with acorresponding extension app to provide content that is viewable within amessage transcript provided by the user interface of each of themessaging apps. In one embodiment, a method can include the followingoperations: receiving, by a first messaging application (app) on a firstdevice, a message and metadata from a second device, the messageincluding content created by a second extension app that operated with asecond messaging app on the second device; launching a first extensionapp, if it is installed on the first device, in response to receivingthe content, the first extension app being identified for launching byan app identifier in the metadata; displaying, after the launching, auser interface of the first extension app within a message containerwithin a message transcript of the first messaging app, wherein the userinterface is configured to provide output, which is dynamic, from thefirst extension app in the message container. In one embodiment, theoutput which is dynamic can change over time and thus it is not static.The change can occur in response to inputs from the user or processesperformed by the first extension app, and the change can occur withoutreceiving any new content from the sending device such as the seconddevice.

In one embodiment, the message container is a message bubble that isspecified by a bubble identifier in the metadata, and the content isassociated with the bubble identifier. In one embodiment, the firstextension app generates the output of the first extension app within themessage container based on the content received from the secondmessaging app. In one embodiment, the user interface of the firstextension app in the message bubble is configured to receive one or moreinputs for the first extension app in addition to receiving an outputfrom the first extension app. In one embodiment, the first and thesecond messaging apps are each configured to transmit short messageservice (SMS) text messages and other content and display the textmessages in message bubbles in the message transcript; in oneembodiment, the first messaging app and the first extension app areconfigured to communicate through an interprocess communication (IPC).In one embodiment, the first messaging app executes in a first sandboxedprocess and the first extension app executes in a second sandboxedprocess that is distinct from the first sandboxed process.

In one embodiment, the first extension app can have a further userinterface which replaces an on-screen keyboard of the first messagingapp after the message bubble containing content generated by the firstextension app receives an input. In one embodiment, the content isdisplayed in the message bubble by the first messaging applicationwithout launching or executing the first extension app, and the contentas displayed before the launching is a snapshot created by the firstextension app or the second extension app. In one embodiment, anapplication program interface (API) can exist between the firstmessaging app and the first extension app. In one embodiment, the firstmessaging app offers to download and install the first extension app ifthe first extension app is not installed.

In one embodiment, the first extension app can modify the content andpass the modified content to the first messaging app, through aninterprocess communication, for transmission to the second messaging appfor delivery to the second extension app on the second device. In oneembodiment, the first extension app can receive a call back from thefirst messaging app which indicates whether the transmission wassuccessful or not. In one embodiment, the first extension app can beconfigured to process the content created by the second extension app,and the second extension app and the first extension app can beidentified by the same app identifier. In one embodiment, the firstextension app can call through an API to the first messaging app torequest a change of the first extension app's view within the firstmessaging app, and this change can be one of (a) a switch from a compactview that replaces the on-screen keyboard of the messaging app to anexpanded view; or (b) a switch from the expanded view to the compactview; or (c) dismissal of the extension app's view. In one embodiment,the message transcript of the first messaging app is viewable when thecompact view of the first extension app is displayed.

In one embodiment, the user interface provided by the first extensionapp in the message bubble is provided through an instance of a remoteview controller that is embedded in the message bubble to allow thefirst extension app to create live dynamic output in the message bubble.In one embodiment, the message transcript can include multiple messagebubbles, some of which use a static, template-based content from anotherextension app operating with the first messaging app. In one embodiment,the live dynamic output changes in response to user inputs in themessage bubble or in a compact view of the first extension app. In oneembodiment, the first extension app can be configured to provide livedynamic content through multiple instances of remote view controllers,with each instance of the multiple instances being associated with acorresponding message bubble in the transcript which shows live dynamicoutput from the first extension app.

In one embodiment, the first extension app can also provide a furtheruser interface in a message staging area or input staging area which isconfigured to receive a message input or other content to be sent; auser typically enters content into the input staging area before it issent and then selects a send command to cause the content to be sent.Hence, in this embodiment, the first extension app can control both theinput into and the output shown in the input staging area in thisembodiment.

In one embodiment, the first messaging app manages a lifecycle of thefirst extension app based on one or more of: (a) an open or closed stateof the messaging app; (b) a switch to a different conversation that doesnot include the message bubble having content generated by the firstextension app; (c) a limit on a number of view controllers for the firstextension app; (d) a limit on a number of view controllers for the firstextension app and any other extension apps in the message transcript; or(e) whether the message bubble containing the content generated by thefirst extension app is on-screen or off-screen. In one embodiment, thelaunching of the first extension app occurs when the message bubblecontaining the content generated by the first extension app is on-screenbut does not occur when the message bubble is off-screen.

In one embodiment, a method can include the following operations:receiving, by a first messaging application (app) on a first device, amessage and metadata from a second device, the message including contentcreated by a second extension app on the second device; determiningwhether a message container currently is on-screen in a messagetranscript of a conversation, the message transcript provided as adisplayed output of the first messaging app, and the message containerincluding an output generated from the content; launching a firstextension app, if it is installed on the first device, in response todetermining the message container is on-screen, the first extension appbeing identified by an app identifier in the metadata; displaying, afterthe launching, a user interface of the first extension app within themessage container in the message transcript.

The methods and systems described herein can be implemented by dataprocessing systems, such as one or more smart phones, tablet computers,desktop computers, laptop computers, smart watches, wearable devices,audio accessories, onboard computers, and other data processing systemsand other consumer electronic devices. The methods and systems describedherein can also be implemented by one or more data processing systemswhich execute executable computer program instructions, stored in one ormore non-transitory machine readable media or medium that cause the oneor more data processing systems to perform the one or more methodsdescribed herein when the program instructions are executed. Thus, theembodiments described herein can include methods, data processingsystems, and non-transitory machine readable media such as DRAM memoryand flash memory.

The above summary does not include an exhaustive list of all embodimentsin this disclosure. All systems and methods can be practiced from allsuitable combinations of the various aspects and embodiments summarizedabove, and also those disclosed in the Detailed Description below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1A shows an example of a user interface for a messaging app on acommunication device.

FIG. 1B shows another example of a user interface for a messaging app ona communication device.

FIG. 2 shows an example of a messaging system which employs one or moremessaging servers to provide a messaging service to a set of clientdevices.

FIG. 3A shows a block diagram of an architecture for providing extensionapps for use with a messaging app according to the one or moreembodiments described herein.

FIG. 3B shows an example of a user interface of a message bubble createdat least in part by an extension app according to one embodiment herein.

FIG. 4A shows an example of a message bubble containing dynamic contentgenerated by an extension app according to one embodiment herein.

FIG. 4B shows an example of the message bubble shown in FIG. 4A after auser has provided an input into the message bubble according to oneembodiment herein.

FIG. 4C shows an example of the user interface of a messaging app whichincludes, in the transcript of the messaging app, two dynamic messagebubbles from two different extension apps that are operating with themessaging app in the embodiment of FIG. 4C.

FIG. 5 shows an example, in block diagram form, of an architecture of amessaging app operating with multiple extension apps through multipleremote view controllers according to one embodiment described herein.

FIG. 6 is a flowchart which illustrates a method according to oneembodiment described herein.

FIG. 7 is a flowchart which illustrates a method according to oneembodiment for controlling the lifecycle of an extension app that cangenerate dynamic content in a message bubble according to oneembodiment.

FIG. 8A shows an example of a message transcript in which a messagebubble containing dynamic content is outside the viewable portion of themessage transcript.

FIG. 8B shows the example of the message transcript shown in FIG. 8Aafter the transcript has been scrolled in a direction to cause themessage bubble containing dynamic content to be viewable in the messagetranscript.

FIG. 9 shows an example of an extension app controlling the inputstaging area in a messaging app according to one embodiment.

FIG. 10 shows an example according to one embodiment in which extensionapps on two or more devices can interact with each other through themessaging apps on each device.

FIG. 11 is an example of a data processing system which can be used withthe one or more embodiments described herein.

DETAILED DESCRIPTION

Various embodiments and aspects will be described with reference todetails discussed below, and the accompanying drawings will illustratethe various embodiments. The following description and drawings areillustrative and are not to be construed as limiting. Numerous specificdetails are described to provide a thorough understanding of variousembodiments. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification do not necessarily all refer to the sameembodiment. The processes depicted in the figures that follow areperformed by processing logic that comprises hardware (e.g. circuitry,dedicated logic, etc.), software, or a combination of both. Although theprocesses are described below in terms of some sequential operations, itshould be appreciated that some of the operations described may beperformed in a different order. Moreover, some operations may beperformed in parallel rather than sequentially.

The various embodiments described herein relate to messaging systemssuch as text messaging systems or “chat” messaging systems or othersystems which allow devices to communicate messages between the devices.For example, iMessage from Apple Inc. of Cupertino, Calif. is an exampleof a messaging service for iOS devices and Mac (OS X) computers.Typically, a messaging system includes the plurality of client devices,each including at least one messaging app, and a set of one or moremessaging servers that can receive messages from client devices andtransmit messages to client devices. FIG. 1A shows an example of a userinterface of a messaging app on a client device. The client device canbe a communication device 10 which can be a smartphone, or tabletcomputer, or a desktop computer or a laptop computer, wearable, on-boardcomputer, or other data processing systems or other consumer electronicsdevices. In one embodiment, the device can include a conventional touchscreen that can both display images and also accept touch inputs from auser. The touch screen 12 on the communication device can display theuser interface of the messaging app which can include a messagetranscript 16 and an on-screen keyboard 20 below the message transcript16. In addition, the user interface of the messaging app can include auser name 14 indicating the recipient, in one embodiment, of messagessent from the communication device 10. In addition, the user interfacecan include a text entry region 18 which indicates the content of thetext entered by the user before it is sent; in a sense, the text entryregion 18 is a text staging area indicating text that is ready to besent to the recipient.

FIG. 1B shows a more detailed example of a user interface of a messagingapp on a communication device 10A. The user interface is displayed inthis embodiment on touch screen 12A and includes on-screen keyboard 20A,text entry region 18A, a message transcript 16A and a user name 14Ashowing the name of the other user(s) to whom messages are sent and fromwhom messages are received. The text entry region 18A is a staging areafor content such as one or more of text, stickers, extension appcontent, images etc. which are ready to be sent in response to theuser's selection of the send command (and in one embodiment, the contentcan be edited in the staging area). In the example shown in FIG. 1B,messages from Freddy (user name 14A) are shown on the left side of themessage transcript 16A and messages sent by the user of communicationdevice 10A are shown on the right side of message transcript 16A. Hence,message bubble 17 shows the message “Where at?” sent by the user ofcommunication device 10A to Freddy as a response to the message fromFreddy “Dinner tonight?”. The message in message bubble 17 causes Freddyto respond with “How's this?” shown in message bubble 19. The use of theword “bubble”, such as message bubble or session bubble, etc. is notmeant to imply any specific shape or form; rather it is intended to meanany shape or form of demarcation between messages between two or moreparticipants, and thus the demarcation can use boxes or lines or messagecontainers or different colors, etc. Thus the phrase “message bubble” ismeant to cover all such demarcations (or other ways to distinguish)between messages between two or more participants, and particularly inthe context of such demarcations or other ways to distinguish in amessage transcript. In one embodiment, the message transcript can bescrolled up and down and messages in the message transcript arepresented in sequence according to their time, thus the user can see theactual chat or conversation over time by scrolling the view up or down.The user interface shown in FIG. 1B also includes three icons 22, 23,and 24 above the on-screen keyboard 20A and to the left of the textentry region 18A. The extension apps view icon 22, when selected, causesthe display of a view of the installed extensions app that can operatewith the messaging app, and the view provided can be a browsable view toallow the user to scroll through multiple pages showing all of theinstalled extension apps that are configured to operate with themessaging app according to the one or more embodiments described herein.The imaging app icon 23 can in one embodiment be an icon that whenselected causes the launching of a plugin of the messaging app thatprovides image creation within the messaging app's process. The cameraapp icon 24 in one embodiment can, when selected, cause thecommunication device 10A to enter into a camera mode in which a cameraof a device can capture still images or video images which can be placedinto the message in order to send the image or video.

A brief overview of an example of a messaging system will now beprovided in conjunction with FIG. 2. A messaging system 50 can include aplurality of client devices, such as client devices 53 and 54. Each ofthese client devices can include at least one messaging app that isconfigured to operate with extension apps according to the one or moreembodiments described herein and also communicate at least text messagesand optionally resource locators or images or other content for devicesthat are not compatible with the extension app architecture in amessaging app. In a typical messaging system, there can be millions ofclient devices communicating through a set of messaging servers. In oneembodiment, a plurality of messaging servers can be configured toreceive encrypted messages from sending devices and then transmit thoseencrypted messages to the receiving devices. Another set of servers canbe configured to receive non-text content, such as images or other“attachments” and provide those images or attachments in downloadoperations to receiving devices in response to requests from thosereceiving devices to obtain the images or attachments. In oneembodiment, a sender's outgoing message is individually encrypted foreach of the receiver's devices. In one embodiment, an asymmetric RSAencryption algorithm can be used to perform the encryption. In oneembodiment, the public RSA encryption keys of each of the receivingdevices can be retrieved from a directory service (maintained by the oneor more messaging servers) which includes a database, such as a database52 which is coupled to the one or more messaging servers 51. When aclient device, such as client device 53 seeks to send a message toanother client device, it identifies the other client device (such as byan email address or a telephone number or other identifier) to the oneor more messaging servers 51. That identifier is sent from the clientdevice, such as client device 53 to one or more messaging servers 51which then perform a lookup operation in the database 52 based upon thesupplied identifier to retrieve the public key that corresponds to thatidentifier. That public key is then transmitted back to the clientdevice which requested the public key for that particular receivingdevice, and then the client device can encrypt the message using thepublic key or using another key (e.g. a symmetric key) which can berandomly generated, and that other key is encrypted with the public RSAencryption key for the particular receiving device. In one embodiment,the randomly generated key can be randomly generated on a per messagebasis. In one embodiment, the resulting messages, one for each receivingdevice, consists of the encrypted message text, the encrypted messagekey, and the sender's digital signature, and this resulting message foreach receiving device is then uploaded to the one or more messagingservers 51 for delivery to the recipient client devices, such as clientdevice 54. In one embodiment, the messaging system 50 can be configuredto operate through “public” networks which include public WiFi accesspoints (such as WiFi access points in coffee shops, airports, etc.) andalso the Internet. The messaging apps on each of the client devices 53and 54 can also be configured to operate with the “private” networksprovided by wireless cellular telephone carriers, such as Verizon andAT&T, and the messaging apps can be configured to seamlessly switchbetween the use of the private and public networks depending upon theavailability of each and also depending upon the compatibility of eachof the client devices in a messaging session. In one embodiment, themessaging servers 51 can include a set of push notification serverswhich receive the uploaded text messages and which “push” those textmessages to receiving devices.

A messaging system in one embodiment on a client device includes amessaging app and one or more extension apps that each operate asseparate processes. In one embodiment, the message app and the one ormore extension apps can each be separate sandboxed processes thatoperate or execute in their own memory spaces. In addition, themessaging app can also operate with plug-ins, such as an image creationplug-in or other plug-ins which operate within the same process andmemory space as the messaging app. The messaging app and each extensionapp communicate with each other through an interprocess communication,such as the XPC framework provided in iOS and Mac OS X. The messagingapp is designed to receive text on a sending device from a user of thedevice and display the text in a message transcript and send the text,through a set of one or more messaging servers, to a receiving devicewhich displays, through a counterpart messaging application on thereceiving device, the received text in a message transcript on thereceiving device. The receiving device and the sending device can eachhave a copy of the same extension app which are configured to createcontent of a certain type depending upon the particular extension app(or, in an alternative embodiment, each can include a copy of differentextension apps that are compatible for the content used by them).

FIG. 3A is an example of a software architecture in which a messagingapp and one or more extension apps operate together to provide anenhanced messaging system. The messaging system shown in FIG. 3Aincludes a messaging app 76, an extension app 83 which can providedynamic content, and a view controller 80. In one embodiment, themessaging app 76 allows for the selection of a conversation from a listof conversations that have occurred in a messaging account used by themessaging app 76. The messaging app 76 can create the message transcriptfor a selected conversation and can also allow for the creation of a newconversation as well as the deletion of existing conversations. In oneembodiment, the messaging app 76 provides for operability throughinterprocess communication (IPC) with one or more extension appsdesigned to operate with the messaging app 76. In one embodiment, themessaging app 76 can execute processes and tasks in its sandbox whilethe extension app, such as extension app 83, can execute processes andtasks in its own sandbox, which is separate and distinct from thesandbox environment of the messaging app 76. In one embodiment, eachextension app can operate within its own sandbox, which can includelimitations on memory which can be accessed by each process as well asthe permissions granted to each process. In one embodiment, theextension apps can be provided with fewer system permissions relative tothe messaging app so that the extension apps operate in a morerestricted environment than the messaging app.

The extension app 83 can be an extension app that generates dynamic,live content for display in, for example, one or more message bubbles inthe transcript. In one embodiment, the extension app 83 can alsogenerate dynamic, live content in an input staging area, such as theinput staging area 153 shown in FIG. 4C and the input staging area 407shown in FIG. 9. The message bubbles that display the dynamic livecontent from the extension app 83 are examples of user interfacesprovided by the extension app 83. Similarly, the output of the extensionapp 83 which is displayed in, for example, the input staging area isanother example of the user interface of the extension app 83. In someinstances, these user interfaces can both display an output as well asreceive inputs, such as touch inputs or other inputs directed toward theuser interface, and these inputs can be received by the extension app 83and processed according to the internal processes of the extension app83. The dynamic, live content is not static and can change over time,and can change based upon inputs from the user on the data processingsystem or based upon inputs from the data processing system or basedupon processes in the extension app 83. In one embodiment, the userinterface of the extension app, such as extension app 83, within abubble or other areas of the messaging app, such as the input stagingarea, can be provided through one or more view controllers such as aremote view controller used in applications that operate on theoperating system iOS from Apple Inc. of Cupertino, Calif. in oneembodiment. In one embodiment, each message bubble can have its owninstance of a remote view controller which provides the user interfaceof the extension app, such as extension app 83. Similarly, the userinterface of an extension app which provides dynamic content for aninput staging area can also have its own instance of a remote viewcontroller for the input staging area. One message transcript for aconversation can have multiple message bubbles from a single dynamicextension app, and the message transcript can also include messagebubbles from other dynamic extension apps. Thus, it is possible for amessage transcript to concurrently require or support multiple viewcontrollers for the multiple message bubbles within the transcript forthe one or more dynamic extension apps. In one embodiment, the viewcontrollers are kept in a cache in memory. In one embodiment, the numberof view controllers can be limited in that cache in order to controlresource usage by the messaging system. In one embodiment, the messagingapp 76 can also work with extension apps that have static contentpresented within one or more message bubbles. Examples of extension appsthat have static content are described in U.S. patent application Ser.No. 15/275,136, filed Sep. 23, 2016, entitled “Messaging ApplicationInteracting with One or More Extension Applications,” which applicationis hereby incorporated herein by reference in its entirety.

In one embodiment, the extension apps which are configured to operatewith a messaging app are not executable outside of the messaging app,and thus their lifecycle is managed entirely by the messaging app.Moreover, as described in U.S. patent application Ser. No. 15/275,136,filed Sep. 23, 2016, the downloading and installing of the extensionapps can be controlled exclusively by the messaging app in oneembodiment.

FIG. 3B shows an example of a message bubble 82 which includes dynamiccontent 85 from a dynamic extension app, such as dynamic extension app83, which is configured to provide a dynamic live user interface formessage bubbles and other items within the messaging app through, forexample, one or more remote view controllers as described above. Thedynamic content 85 can include an icon 87 which can be the icon of theextension app which created the dynamic content 85. Moreover, themessage bubble 82 can also include a text message 88, which is optional,and which contains text from a source such as the sender of the messageor the messaging app or extension app on the sender's device.

FIGS. 4A and 4B show an example of a messaging app which operates with adynamic extension app in order to provide dynamic, live content in oneor more message bubbles within the message transcript of theconversation provided by the messaging app. The transcript 101 includesthree message bubbles which are labeled as message bubble 102, messagebubble 104, and message bubble 106. In one embodiment, message bubbles102 and 106 can be generated by a dynamic extension app, such as theextension app 83 shown in FIG. 3A. Message bubble 104 can display textentered by the user of the device which is executing the messaging appthat displays the transcript 101. In one embodiment, the sender ofcontent in message bubble 102 and 106 can be a restaurant, which has itsown extension app installed on the user's device, and this extension appcan generate the dynamic live content in message bubble 106. In oneembodiment, the output of the message bubble can change from messagebubble 106 shown in FIG. 4A to message bubble 106A shown in FIG. 4B inresponse to an input from the user. For example, in one embodiment, theuser may tap on message bubble 106 which causes the presentation of aslider user interface from which a user can select a number of seats toreserve. Alternatively, other known user interfaces for selecting anumber could also be used (such as an on-screen keyboard). In any case,the user's interaction with the user interface through message bubble106 can allow a user to enter an input into the message bubble 106, andthat input can then be provided to the dynamic extension app which canthen provide data based upon that input back to the sender of messagebubble 106 (for example, see FIG. 10). In an alternative embodiment, acompact view of the extension app that generates the dynamic contentshown in message bubble 106 can receive the input from the user, whichthen causes the output of the message bubble 106A to be displayed whichreflects a change relative to message bubble 106.

It will be appreciated that many different inputs can be received withinthe user interface generated by the dynamic extension app, and manydifferent types of outputs can be in response to those inputs generatedand displayed within a dynamic message bubble or other regions of theuser interface of the messaging app.

FIG. 4C shows an example of a user interface of a messaging app 150which displays a transcript 151 of the messages in a conversation. Theviewable portion of this transcript includes message bubbles 157 from afirst extension app and message bubble 159 from a second extension app.In one embodiment, the message app which generates the content ofmessage bubble 157 can be a dynamic extension app, and the extension appwhich generates the dynamic content of message bubble 159 can also be adynamic extension app. Thus, both message bubbles 157 and 159 can bedynamic live message bubbles in which the content can change undercontrol of the respective extension app. Moreover, message bubble 157and message bubble 159 can each receive inputs and provide those inputsto their respective dynamic extension apps. For example, the restaurantreview shown in message bubble 157 can receive user inputs indicatingthe user's thoughts or comments or review of a particular restaurant.The user interface provided by the extension app which displays dynamiccontent within message bubble 159 can also receive inputs to reserveseats at a particular restaurant. In the example shown in FIG. 4C, theextension app for restaurant reservation includes its compact view 155in the area where the on-screen keyboard is normally displayed. In thisexample, user input can be received in either the compact view 155 orwithin message bubble 159 in order to create a reservation at arestaurant. In another embodiment, the extension app for a restaurantreservation which controls the compact view 155 and which also controlsthe message bubble 159 can also generate a user interface within theinput staging area 153, and FIG. 9 shows an example of such a userinterface in an input staging area.

FIG. 5 shows an example of a software architecture which can be used toimplement the messaging system shown in FIG. 4C. In the example shown inFIG. 5, the messaging app 201 allows two different extension apps tocontrol at least one message bubble in a message transcript of aconversation. In the example shown in FIG. 5, dynamic extension app 203uses a remote view controller 204 to generate a dynamic user interfacewithin the message bubble 208. The dynamic extension app 205 has oneinstance of a remote view controller for each message bubble in thetranscript. In particular, extension app 205 uses remote view controller211 to provide a dynamic user interface within message bubble 212, andalso uses the remote view controller 206 to provide a dynamic userinterface within message bubble 210. In one embodiment, a separateinstantiation or instance of a remote view controller is provided foreach and every message bubble for a particular extension app thatprovides content to those message bubbles.

FIG. 6 shows an example of a method according to one embodiment whichuses a dynamic extension app to generate a dynamic user interfacewithin, for example, a message bubble in a transcript of a messagingapp. The method shown in FIG. 6 can be performed with the softwarearchitecture shown in FIG. 3A or the architecture shown in FIG. 5 andcan result in the display of the transcripts shown in FIGS. 4A, 4B, and4C. In operation 251, a first messaging app on a first device receives amessage and metadata from a second device, where the message includescontent created by a second extension app that can be a dynamicextension app that operated with a second messaging app on the seconddevice. In one embodiment, the second device can be a sender's devicethat has sent the message and metadata from the sender's device to thefirst device which uses the first messaging app to receive the messageand metadata. In operation 253, a first extension app can be launched ifit is installed on the first device in response to receiving thecontent. In one embodiment, the metadata received from the second devicecan identify the first extension app, and that metadata can be used todetermine whether the first extension app is installed on the firstdevice. If the first extension app is not installed, the first messagingapp on the first device can offer to download and install the firstextension app from an extension app store which can be hosted within thefirst messaging app as described in U.S. patent application Ser. No.15/275,136, filed Sep. 23, 2016. In one embodiment, the first extensionapp may be launched only if the message bubble containing contentgenerated by the first extension app is on-screen within the displayedtranscript. In other words, launching of the first extension app may bedeferred until it is visible and on-screen in the message transcript. Inoperation 255, a dynamic user interface of the first extension app canbe displayed, after launching, within a message container such as amessage bubble within the message transcript provided by the firstmessaging app. The dynamic user interface provided in operation 255 canshow an output, which is dynamic, from the first extension app. Then, inoperation 257, the first extension app can receive inputs and change theoutput within the message bubble based upon the inputs. An example ofthis change is shown in FIGS. 4A and 4B.

FIG. 7 shows an example of lifecycle management for a dynamic extensionapp according to one embodiment. The method can begin in operation 301,in which a user selects a conversation from a list of conversationsprovided by the messaging app. The selected conversation is opened bydisplaying a message transcript of the conversation. The messagetranscript contains a message bubble with received content for use by adynamic extension app, such as dynamic extension app 83 shown in FIG.3A. The extension app can be identified by metadata so that it can belaunched. In one embodiment, the message bubble containing the dynamiccontent may have been received many days ago, and hence may not beviewable within the on-screen message transcript. The state ofviewability is tested in operation 303 by determining whether themessage bubble which contains dynamic content generated by a dynamicextension app is on-screen. If operation 303 determines that the dynamicmessage bubble is not on-screen, then the extension app identified bythe metadata is not launched. This can be seen in FIGS. 8A and 8B, whichshows a messaging app 350 that displays a transcript 351. In thesituation shown in FIG. 8A, message bubble 353 is outside of theviewable area of the transcript 351. Message bubble 353 contains dynamiccontent generated by a dynamic extension app, such as dynamic extensionapp 83. In the example shown in FIG. 7, operation 303 will deferlaunching the extension app which generated the dynamic content withinmessage bubble 353 until the message bubble 353 appears within theviewable portion of the transcript 351. This can occur when the userscrolls the transcript down to make the message bubble 353 viewablewithin the on-screen portion of the transcript 351 as shown in FIG. 8B.The result of scrolling down the transcript causes the message bubbles355 and 357 to move down the screen and also causes the message bubble359 to move off of the viewable portion of the transcript 351 as shownin FIG. 8B. If operation 303 determines the message bubble is viewable,then operation 305 can follow in which the messaging app uses a storedsnapshot, which can be a static image of the dynamic content toinitially display content in the message bubble before the dynamicextension app finishes launching. In one embodiment, the stored snapshotcan be created by the first extension app during a prior execution ofthe first extension app (e.g., when the conversation was previouslyopened and the message bubble was displayed then by the first extensionapp in the conversation and the snapshot was created then). In anotherembodiment, the stored snapshot can be created by the second extensionapp (on the sender's device) and sent to the receiver's device. Inoperation 307, the dynamic extension app can be launched while thestatic snapshot is displayed within the message bubble. In operation309, an instance of a view controller, such as a remote view controllerfor the message bubble, is created so that the extension app cangenerate the dynamic user interface for the message bubble. Then inoperation 311, the extension app displays the dynamic content in themessage bubble. At this point, the user can interact with the messagebubble, and FIGS. 4A and 4B provide an example of such a user'sinteraction with the dynamic content in a message bubble having adynamic user interface generated by the dynamic extension app. At thispoint, as shown in operation 313, the messaging app can control thelifecycle of the extension app based upon one or more events or factors.For example, if the user switches to another conversation, the dynamicextension app can be terminated or quit. Similarly, if the messaging appis closed or quit, then the dynamic extension app can be terminated orquit. In one embodiment, if the messaging app is caused to go from theforeground process to a background process, then the dynamic extensionapp can be terminated. In one embodiment, the dynamic extension app canbe terminated if there is memory pressure (such as a low DRAM memorysituation). Also, if the message bubble having the dynamic content isnot on-screen and the number of cache to view controllers has exceeded apredetermined limit, then the dynamic extension app can be quit orterminated.

FIG. 9 shows an example of a user interface of a messaging app in whicha dynamic extension app can generate a dynamic user interface within theinput staging area, such as input staging area 407. The messaging app401 displays a transcript 403 which includes one or more messagebubbles. At the bottom of the transcript, the input staging area 407 canbe selected by the user to present a user interface of an extension appthat is dynamic. In particular, user interface 409 is a dynamic userinterface generated by a dynamic extension app that allows the userinterface 409 to receive inputs for the dynamic extension app and toprovide outputs from the dynamic extension app within the user interface409 in the input staging area 407. In addition, the dynamic extensionapp includes the user interface in the compact view 405 which is afurther user interface region controlled by the dynamic extension app.

FIG. 10 shows an example of the interaction in a session between twoextension apps 505 and 507 on two different devices 501 and 503, wherethe extension apps can both be dynamic extension apps. A first device501 includes a first messaging app 509 and a first dynamic extension app505, and a second device 503 includes a second messaging app 511 and asecond dynamic extension app 507. The first dynamic extension app 505and the second dynamic extension app 507 can have the same appidentifier or metadata indicating the apps are the same but differentinstances of the same app on the two devices. The first device 501 canalso include a remote view controller 515 allowing the first dynamicextension app 505 to provide dynamic user interfaces within messagebubbles or other items within the messaging app 509. Similarly, thesecond device 503 can include a remote view controller 517 to allow thesecond dynamic extension app 507 to provide dynamic user interfaceswithin message bubbles or other components within the second messagingapp 511. The first and second messaging applications can receive dynamiccontent from their respective dynamic extension apps and transmit themto the other device (through one or more messaging servers 521) whichcan provide a session between the two dynamic extension apps. FIG. 4Bshows an example in which a receiving device has received dynamiccontent from a remote dynamic extension app and the user (using a localdynamic extension app) has modified the content by entering a numberwithin the dynamic content of the message bubble which can then be sentback (e.g., as URL, data, etc. 519) to the remote extension app throughthe two messaging apps or through a separate communication channel. Forexample, in one embodiment, in addition to using the two messaging apps509 and 511 to exchange data between the two extension apps 505 and 507,the two extension apps 505 and 507 can open a separate and distinctcommunication channel that does not use the two messaging apps toexchange data. Thus, the URL, data, etc. 519 can be transmitted throughthe separate and distinct communication channel; moreover, eachextension app may communicate with a web server controlled by thedeveloper of the extension apps, etc. and this communication can bethrough the separate and distinct communication channel. Furtherinformation in connection with such sessions between extension apps canbe found in U.S. patent application Ser. No. 15/275,136, filed Sep. 23,2016.

The systems and methods described herein can be implemented in a varietyof different data processing systems and devices, includinggeneral-purpose computer systems, special purpose computer systems, or ahybrid of general purpose and special purpose computer systems.Exemplary data processing systems that can use any one of the methodsdescribed herein include server systems, desktop computers, laptopcomputers, tablet computers, smart phones, cellular telephones, personaldigital assistants (PDAs), embedded electronic devices, or otherconsumer electronic devices.

FIG. 11 is a block diagram of data processing system hardware accordingto an embodiment. Note that while FIG. 11 illustrates the variouscomponents of a data processing system that may be incorporated into amobile or handheld device or other electronic device, it is not intendedto represent any particular architecture or manner of interconnectingthe components as such details are not germane to the present invention.It will also be appreciated that other types of data processing systemsthat have fewer components than shown or more components than shown inFIG. 11 can also be used with the present invention.

As shown in FIG. 11, the data processing system includes one or morebuses 1309 that serve to interconnect the various components of thesystem. One or more processors 1303 are coupled to the one or more buses1309 as is known in the art. Memory 1305 may be DRAM or non-volatile RAMor may be flash memory or other types of memory or a combination of suchmemory devices. This memory is coupled to the one or more buses 1309using techniques known in the art. The data processing system can alsoinclude non-volatile memory 1307, which may be a hard disk drive or aflash memory or a magnetic optical drive or magnetic memory or anoptical drive or other types of memory systems (e.g., ROM) that maintaindata even after power is removed from the system. The non-volatilememory 1307 and the memory 1305 are both coupled to the one or morebuses 1309 using known interfaces and connection techniques. A displaycontroller 1322 is coupled to the one or more buses 1309 in order toreceive display data to be displayed on a display device 1323. Thedisplay device 1323 can include an integrated touch input to provide atouch screen. The data processing system can also include one or moreinput/output (I/O) controllers 1315 which provide interfaces for one ormore I/O devices, such as one or more mice, touch screens, touch pads,joysticks, and other input devices including those known in the art andoutput devices (e.g. speakers). The input/output devices 1317 arecoupled through one or more I/O controllers 1315 as is known in the art.

While FIG. 11 shows that the non-volatile memory 1307 and the memory1305 are coupled to the one or more buses directly rather than through anetwork interface, it will be appreciated that the present invention canutilize non-volatile memory that is remote from the system, such as anetwork storage device which is coupled to the data processing systemthrough a network interface such as a modem or Ethernet interface. Thebuses 1309 can be connected to each other through various bridges,controllers and/or adapters as is well known in the art. In oneembodiment the I/O controller 1315 includes one or more of a USB(Universal Serial Bus) adapter for controlling USB peripherals, an IEEE1394 controller for IEEE 1394 compliant peripherals, or a Thunderboltcontroller for controlling Thunderbolt peripherals. In one embodiment,one or more network device(s) 1325 can be coupled to the bus(es) 1309.The network device(s) 1325 can be wired network devices (e.g., Ethernet)or wireless network devices (e.g., WiFi, Bluetooth).

It will be apparent from this description that aspects of the presentinvention may be embodied, at least in part, in software. That is, thetechniques may be carried out in a data processing system in response toits processor executing a sequence of instructions contained in astorage medium, such as a non-transitory machine-readable storage medium(e.g. volatile DRAM or non-volatile flash memory). In variousembodiments, hardwired circuitry may be used in combination withsoftware instructions to implement the present invention. Thus thetechniques are not limited to any specific combination of hardwarecircuitry and software, or to any particular source for the instructionsexecuted by the data processing system. Moreover, it will be understoodthat where mobile or handheld devices are described, the descriptionencompasses mobile devices (e.g., laptop devices, tablet devices),speaker systems with integrated computing capabilities, handheld devices(e.g., smartphones), as well as embedded systems suitable for use inwearable electronic devices.

In the foregoing specification, specific exemplary embodiments have beendescribed. It will be evident that various modifications may be made tothose embodiments without departing from the broader spirit and scopeset forth in the following claims. The specification and drawings are,accordingly, to be regarded in an illustrative sense rather than arestrictive sense.

The following Appendix includes an embodiment of a set of applicationprogramming interfaces (API) that allow a set of dynamic extension appsto operate with a messaging app to add dynamic, live user interfacesgenerated and used by the dynamic extension apps within message bubblesin a transcript of a conversation and other components presented by themessaging app.

1. A method comprising: receiving, by an application on a device,content corresponding to an extension application; displaying a userinterface of the extension application in a container within a view ofthe application, the user interface comprising the content; anddisplaying an expanded view of the user interface in response to a userinput with respect to the container, wherein the expanded view of theuser interface overlaps at least a portion of the view external to thecontainer and the user interface is configured to provide dynamic outputfrom the extension application.
 2. The method of claim 1, wherein thecontent corresponding to the extension application is received by theapplication via an inter-process communication.
 3. The method of claim1, wherein receiving, by the application on the device, the contentcorresponding to the extension application further comprises: receiving,by the application on the device, metadata and the content correspondingto the extension application.
 4. The method of claim 3, wherein thecontent is associated with the metadata.
 5. The method of claim 4,wherein the extension application is identified by an extensionapplication identifier in the metadata.
 6. The method of claim 1,wherein the expanded view of the user interface comprises a selector forreceiving a user selection.
 7. The method of claim 6, wherein the userselection corresponds to the content.
 8. A device comprising: a memory;and at least one processor configured to: obtain, by an application on adevice, a message including content corresponding to an extensionapplication; display a user interface of the extension application in acontainer within a view of the application, the user interfacecomprising the content; and display an expanded view of the userinterface in response to a user input with respect to the container,wherein the expanded view of the user interface overlaps at least aportion of the view external to the container and the user interface isconfigured to provide dynamic output from the extension application. 9.The device of claim 8, wherein the content corresponding to theextension application is received via an inter-process communication.10. The device of claim 8, wherein the at least one processor is furtherconfigured to: receive, by the application on the device, metadata andthe message including the content corresponding to the extensionapplication.
 11. The device of claim 10, wherein the content isassociated with the metadata.
 12. The device of claim 11, wherein theextension application is identified by an extension applicationidentifier in the metadata.
 13. The device of claim 8, wherein theexpanded view of the user interface comprises a selector for receiving auser selection.
 14. The device of claim 13, wherein the user selectioncorresponds to the content.
 15. A non-transitory machine-readable mediumcomprises code that, when executed by one or more processors, causes theone or more processors to: display a user interface of an extensionapplication in a container within a view of an application on a device,the user interface comprising content; and display an expanded view ofthe user interface in response to a user input with respect to thecontainer, wherein the expanded view of the user interface overlaps atleast a portion of the view external to the container and the userinterface is configured to provide dynamic output from the extensionapplication.
 16. The non-transitory machine-readable medium of claim 15,wherein the content corresponding to the extension application isreceived via an inter-process communication.
 17. The non-transitorymachine-readable medium of claim 15, the code, when executed by the oneor more processors, further causes the one or more processors to:receive, by the application on the device, the content corresponding tothe extension application and metadata.
 18. The non-transitorymachine-readable medium of claim 17, wherein the content is associatedwith the metadata.
 19. The non-transitory machine-readable medium ofclaim 18, wherein the extension application is identified by anextension application identifier in the metadata.
 20. The non-transitorymachine-readable medium of claim 15, wherein the expanded view of theuser interface comprises a selector for receiving a user selection.